MCG-ICT at MediaEval 2016 Verifying Tweets from both Text and Visual Content

The Verifying Multimedia Use Task aims to automatically detect manipulated and fake web multimedia content. We have two important improvements this year: On the one hand, considering that the prediction based on a short tweet is unreliable, we propose a topic-level credibility prediction framework. This framework exploits the internal relations of tweets belonging to same topic. Besides, we enhance the prediction precision of the framework by sampling topics and exploring topic-level features. On the other hand, motivated by the idea that manually edited or low-quality videos tend to be fake, we reference the handbook[1] about detecting manual editions and build a decision tree on videos. 1. PROPOSED APPROACH We treat the task as a binary classification problem: real or fake. Generally, a tweet contains two kinds of content: text content and visual content. So, we build two classification models respectively: for text content, the task pays more attention to small events than breaking news this year. More than 59% events contain less than 10 tweets and 95% are less than 50 tweets. Compared with last year’s 42.5 tweets per event, the small event verification is more challenging. We propose a topic-level verification framework, and improve its performance by exploring topic-level features and sampling on topics. For visual content, we reference the handbook[1] about detecting manual editions and build a decision tree on videos. However, the task focuses only on detecting fake tweets while we put efforts to identify on both categories. Finally, we propose a method performing relatively well on both real and fake tweets. More details about the task can be found in [2]. 1.1 Text Analysis Approach As illustrated in Figure 1, the framework of our text analysis approach consists of three parts: a message-level classifier, a topic-level classifier and a fusing part. Like many traditional text analysis methods, we firstly build a messagelevel classifier based on the given content features and user features. However, a tweet is very short (no more than 140 words) and its meaning is incomplete. The credibility prediction on message-level is unreliable. We observe that each tweet contains videos/images in Copyright is held by the author/owner(s). MediaEval 2016 Workshop, Oct. 20-21, 2016, Hilversum, Netherlands. Figure 1: The framework of the text analysis approach our data, and tweets containing the same videos/images are rather independent but have strong relations with each other. More specifically, they attend to have same credibilities. In order to exploit their inner-relations, we take the tweets of the same video/image as a topic, and build a topic-level classifier. Compared with an independent tweet, a topic can maintain principal information and also eliminate random noise. As the primary contribution in the text analysis, the topic-level improves the F1 value of 4% on fake tweets and more than 8% on real tweets. Two main innovations of the topic-level part are as follows: Topic-level Features Extracting: For each topic, we compute the average of its tweets’ features as its features. Besides, we propose several statistic features which are listed in Table 1. Combining the two kinds of features above, we finally get the whole topic-level features. It turns out that these statistic features are quite effective for identifying fake tweets. They boost the topic-level classifier’s F1 value on fake tweets by more than 14%. Topics Sampling: In our dataset, More than 59% topics contain less than 10 tweets and 95% are less than 50 tweets, which means there are quite a few small topics. To remove the noise brought by these small topics, we sample topics with high confidence in a 10-fold cross validation process. The sampling keeps the balance between fake and real Table 1: Topic Layer New Statistic Features